Digital display of videos of scenes is a useful and commonly-practiced technique. Videos typically include a series of individual frames at a selected frame rate, e.g. 24 frames per second (fps), and each frame represents light accumulated for a selected exposure time, e.g. 41.7 ms (= 1/24 sec.). Each frame comprises a two-dimensional array of individual pixels.
Scenes, and thus videos of scenes, can contain global motion or local motion. Global motion refers to relative motion between an image capture device and the scene being imaged, such as when a camera is panning across a scene. Local motion refers to motion of objects within the scene, such as a ball being thrown. When a video contains either global motion or local motion rapid enough to cause features within the image to move across more than one pixel during the exposure time of a frame, image quality can be degraded by blurring and smearing of the image. It is therefore desirable to reduce blur of rapidly moving objects in video frames to improve image quality.
Typically higher quality video such as high definition or HD video (720p, 1080i or 1080p) is captured at a higher frame rate, 30 fps or 60 fps, to reduce the blurring associated with motion during capture. However, when rapid motion is present in a scene, such as a ball being thrown in a sporting event, the image of the ball can be noticeably blurred even at 60 fps. Very fast frame rates can be used to reduce blur and improve video image quality of rapidly moving objects. However, as the frame rate is increased, the amount of image data increases proportionately, which can result in data rates too high for data storage, image processing or data transmission bandwidth in imaging systems such as a consumer video camera, a digital camera or a cell phone camera.
One factor in blur of frames on a display is the response time of the display, the time required for the light from a particular pixel to change from one value to the next. Electroluminescent (EL) displays, such as organic light emitting diode (OLED) displays, employ materials that have response times that are measured in microseconds (Samsung reports a response time for their active matrix OLED display of 10 microseconds on their website: http://www.samsungsdi.com/contents/en/product/oled/type01.html). As used herein, the “update rate” of a display is the rate at which the commanded light output of the display is changed. For example, a conventional 24 fps display has a 24 Hz update rate: each time a new frame is received, the display is commanded to output light corresponding to that frame. The update rate is limited by the response time of the display. The low response times of EL displays thus make them theoretically capable of achieving very high update rates
However, limitations in the drive electronics of EL displays, the image transmission systems and the image processing system within the video capture devices such as consumer video cameras, a digital camera or a cell phone camera, do not permit update rates above approximately 30 Hz to be supported at higher resolutions. If the capture systems supported very high frame rate with high resolution video capture, such video would require large amounts of bandwidth for transmission and special electronic drives for the display.
Video compression techniques such as described in U.S. Pat. Nos. 6,931,065 and 5,969,764 ('764) are useful in reducing the transmission bandwidth of images, partly based on detecting changes (mean absolute differences) between frames and avoiding the transmission of duplicate image data for multiple frames when the scene is not changing, to reduce data transmission bandwidth and data storage requirements. As such, the technique described in the '764 patent effectively reduces the data transmission rate for regions of the scene that are not changing and keeps the original data transmission rate for areas where motion is present. However, on decompression, video images are reconstructed for display at a constant update rate. As such, this method does not change the display update rate.
U.S. Pat. No. 5,389,965 describes a variable frame rate system for video communication. This system permits the user to select the frame rate used to deliver the desired image quality in a mobile communication environment where data transmission bit rates are limited. Slow frame rates are used to deliver higher resolution images at the expense of jerky motion. Faster frame rates deliver smoother motion with lower resolution images. However, the frame rate is constant within the frame rate selected by the user so as such, the frame rate does not change in response to the motion present in the image being displayed. Further, Cok in commonly-assigned U.S. Pat. No. 7,242,850 provides a method for providing a variable frame rate in a display system where the frame rate is dependent upon the motion within the scene as it was originally captured. This patent discusses changing the rate at which entire frames are delivered within the video sequence and requires the system be capable of adapting to the increased bandwidth required to deliver the higher frame rates during rapid motion.
Within the display literature, it is known to receive video at 60 fps and to up-convert the input video to 120 fps to provide an update rate of 120 Hz to display the video without artifacts. For example, Shin et al. in a paper entitled “Motion Interpolation Performance of 120 Hz Display Systems” published in the SID '08 Digest (2008) discusses producing 120 fps video using interpolation to improve motion blur and judder. By upconverting the input signal within the display device, the video can be displayed at a faster frame rate. However, the display and drivers must be designed to support relatively high-resolution updates at a full 120 Hz. Doubling the rate of such drivers from 60 Hz to 120 Hz can be expensive. Furthermore, in displays such as OLED displays, which typically have a high capacitance and have drive lines with a significant resistance, accurately updating information at 120 Hz and maintaining the full bit depth of the display can present a significant challenge. Further, upconversion to 120 fps does not recognize the problem that significant image blur can be introduced during motion capture, and this motion blur is not reduced through these schemes. This is described in Klompenhouwer (2007), “Dynamic Resolution: Motion Blur from Display and Camera,” SID 2007 Digest. As described in this paper, it is important that the image be provided with a short temporal aperture as well as it is important that the display provide a short temporal aperture.
Hekstra et al., in U.S. Patent Application Publication No. 2005/0168492, disclose reducing image blur by decreasing the hold time of the display. Decreasing the hold time increases the update rate, as the display is set to output light corresponding to an image at the frame rate, e.g. 60 Hz, and set to output no light at 60 Hz approximately 180 degrees out of phase with the first updates, resulting in a total update rate of 120 Hz. According to this scheme, an input video can be analyzed to determine the rate of motion and the hold time of a display can be controlled based upon the determined rate of motion. However, this method requires an estimation of the rate of motion directly from the video, which is typically input at 60 fps. Since this video is captured at this rate, it has likely undergone motion blur during capture and this blurring can make the estimation of motion difficult. Therefore, the resulting video will contain artifacts due to errors in estimation of the rate of motion. Further, this method ignores the fact that significant image blur can be introduced during motion capture, and this motion blur is not reduced through this method. Consequently, there exists a need for an improved method for the display of video image data for rapidly moving objects in a way that does not substantially increase the amount of video image data to be processed or substantially increase the bandwidth of the display drivers.